When mma_inst_tile_k > 1, cute.gemm() generates multiple sub-MMA
instructions that all share the same ACCUMULATE flag. With
ACCUMULATE=False on the first K tile, every sub-MMA cleared the
accumulator so only the last sub-MMA's result survived, losing
(mma_inst_tile_k - 1) * mma_inst_shape_k elements per output tile.

This caused GSM8K accuracy to drop from 64.7% to 28.5%.

Fix by adding an inner kblock loop that iterates sub-MMA instructions
individually and sets ACCUMULATE=True after the first cute.gemm() call,
matching the pattern used by blockscaled_contiguous_grouped_gemm.py.

GSM8K accuracy restored to 64.86% (reference: 64.74%).

Signed-off-by: peaceh <103117813+peaceh-nv@users.noreply.github.com>

…kwell

Add use_cute_dsl_bf16_gemm flag to enable CuTe DSL BF16 persistent GEMM
for unquantized Linear layers in MLA attention (kv_a_proj_with_mqa,
q_b_proj, kv_b_proj). This complements the existing BF16 BMM support.

Changes:
- Add CuteDSLBf16BlackwellGemmRunner class and custom op in cute_dsl_custom_ops.py
- Add use_cute_dsl_bf16_gemm parameter to Linear class and UnquantizedLinearMethod
- Wire use_cute_dsl_bf16_gemm through ModelConfig, LlmArgs, and model_loader
- Pass flag to MLA Linear layers in attention.py
- Add --use_cute_dsl_bf16_gemm CLI argument to quickstart_advanced.py
- Add integration tests for single GPU and 4 GPU configurations

Signed-off-by: Pei He <peih@nvidia.com>
Signed-off-by: peaceh <103117813+peaceh-nv@users.noreply.github.com>

…GEMM (FP32 output)

Enable CuTe DSL BF16 GEMM kernel for DeepseekV3Gate router GEMM on Blackwell.
The router computes BF16 input @ BF16 weight -> FP32 logits, which our
persistent GEMM kernel already supports via FP32 accumulator and FP32 output.

Key changes:
- Support FP32 output dtype in CuteDSLBf16BlackwellGemmRunner (detect from
  output tensor instead of hardcoding BF16, add c_dtype to kernel cache key)
- Relax cute_dsl_bf16_gemm_blackwell custom op to accept BF16 or FP32 output
- Add CuTe DSL dispatch in DeepseekV3Gate.forward() gated by
  use_cute_dsl_bf16_gemm flag, with fallback to dsv3_router_gemm_op

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: peaceh <103117813+peaceh-nv@users.noreply.github.com>

… path

Add wrapper_strided to PersistentDenseGemmKernel that accepts explicit A
tensor strides, enabling non-contiguous views (e.g. from .transpose()) to
be passed directly to TMA without .contiguous() copies. Update the BMM
runner to compute and pass A strides instead of forcing contiguous tensors,
removing the direct_copy_kernel_cuda overhead between attention and BMM.

Signed-off-by: peaceh <103117813+peaceh-nv@users.noreply.github.com>

…ction_core

Signed-off-by: peaceh <103117813+peaceh-nv@users.noreply.github.com>

…ed IDs

The previous entries lacked pytest parameter brackets, which wouldn't
match actual test node IDs. Expand to all 12 parametrized variants.

Signed-off-by: peaceh <103117813+peaceh-nv@users.noreply.github.com>

…BMM code

Signed-off-by: peaceh <103117813+peaceh-nv@users.noreply.github.com>

Signed-off-by: peaceh <103117813+peaceh-nv@users.noreply.github.com>

Apply ruff format/lint fixes:
- Convert multi-line docstrings to single-line where appropriate (D200)
- Remove f-string prefix on strings without placeholders (F541)
- Remove unused import
- Use consistent double-quote docstrings instead of single-quotes
- Fix indentation in docstrings

Signed-off-by: Peace He <103117813+peaceh-nv@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Signed-off-by: JacobHu-NV <266902545+JacobHu-NV@users.noreply.github.com>

Signed-off-by: JacobHu-NV <266902545+JacobHu-NV@users.noreply.github.com>

…P, and no-overlap strategies

Refactor fused_moe_densegemm.py into three dedicated modules:
- fused_moe_densegemm_gc.py: GreenContext-based overlap
- fused_moe_densegemm_smp.py: SM-partitioned overlap
- fused_moe_densegemm_no_overlap.py: no-overlap baseline

Update configurable_moe.py, modeling_deepseekv3.py, cute_dsl_custom_ops.py,
fc2.py, and utils.py accordingly.

Signed-off-by: JacobHu-NV <266902545+JacobHu-NV@users.noreply.github.com>